Imaging input/output with shared spatial modulator

ABSTRACT

An image input/output apparatus and method includes a light source in optical communication with a spatial radiation modulator for projecting an output image and an image sensor for capturing an input image along a shared input/output path. In a described embodiment, the display of an output image and the capture of an input image is effected using a common spatial radiation modulator (SRM) in the form of a deformable mirror device (DMD).

RELATED PATENT APPLICATIONS

This application is a continuation-in-part of U.S. application Ser. No.12/608,646 (TI-67274) filed Oct. 29, 2009; which claims the benefit ofU.S. Provisional Application No. 61/109,349 (TI-67274PS) filed Oct. 29,2008; and which is a continuation-in-part of U.S. application Ser. No.12/493,865 (TI-64800) filed Jun. 29, 2009, which claims the benefit ofU.S. Provisional Application No. 61/076,536 (TI-64800PS) filed Jun. 27,2008; the entireties of all of which are also incorporated herein byreference.

This application is a continuation-in-part of U.S. application Ser. No.12/334,240 (TI-65497) filed Dec. 12, 2008; the entirety of which isincorporated herein by reference.

This application also claims the benefit of U.S. Provisional ApplicationNo. 61/161,912 (TI-67001) filed Mar. 20, 2009; U.S. ProvisionalApplication No. 61/162,048 (TI-67002) filed Mar. 20, 2009; U.S.Provisional Application No. 61/162,508 (TI-67394) filed Mar. 23, 2009;U.S. Provisional Application No. 61/162,668 (TI-66605) filed Mar. 23,2009; and U.S. Provisional Application No. 61/165,353 (TI-67927) filedMar. 31, 2009; the entireties of all of which are also incorporatedherein by reference.

TECHNICAL FIELD

This invention relates to methods and devices for image output, such asfor projection of images onto image display surfaces; and, morespecifically, to such methods and devices that use shared resources toalso include capability for image input, such as for recording of imagesby cameras or other image sensing devices utilizing shared optical orother radiation paths for image input and output. The invention findsparticular application in systems that utilize spatial light modulators,such as deformable micromirror devices, for sharing image input andoutput radiation paths.

BACKGROUND

Spatial light modulators (SLMs) are devices used to control thedistribution of light in an optical system. SLMs are typicallyconfigured as one or two-dimensional arrays of individually addressableoptical elements, representing pixels of an image. These elements modifyeither the amplitude or the phase of the light distribution within theoptical system.

SLMs can be divided into various types, including an electro-optic,magneto-optic, liquid crystal, and deformable mirror devices. Thesedifferent types may be further characterized according to whether theyare suitable for amplitude modulation, phase modulation, etc.

SLMs and their applications are described in various patents and patentapplications. An example visual display system having a spatial lightmodulator with individually controllable elements, wherein each elementis capable of producing an individual light beam directed toward adisplay surface, is described in U.S. Pat. No. 5,214,420, entitled“Spatial Light Modulator Projection System with Random Polarity Light.”An example method and structure for providing system control to aspatial light modulator display is described in U.S. Pat. No. 5,254,980,entitled “DMD Display System Controller.” The entireties of both patentsare incorporated by reference herein.

Many applications involve using SLMs in display systems, where an SLMoptics unit replaces a raster scan unit. These are image outputgeneration systems, in which the SLM receives input data in the form ofelectrical signals for the purpose of determining how light is to bereflected by its pixel elements as light output to a display screen,printer, or other such equipment. Thus, SLMs are traditionally used totransform an electrical signal to light patterns, and thereby generatean image output.

Conventional means for creating an electrical output signal from acaptured input image as an electronic signal, as opposed to recreatingan output image from an input signal, do not typically involve the useof SLMs. In many applications the light receiving device is aphotosensor array, in which an array of photosensor elements is used todifferentiate pixels of the image. Each element of the array generates asignal corresponding to a pixel point of the image. This signal can betransmitted, digitized, or otherwise processed for reconstitution intoan image at a desired time and place.

A system for capturing an input image using SLM optics with asingle-element sensor is described in U.S. Pat. No. 5,212,555, entitled“Image Capture with Spatial Light Modulator and Single-CellPhotosensor”; the entirety of which is incorporated herein by reference.In this arrangement, as each pixel element in an SLM array isindividually switched into a light reflecting position, light from therespective pixel is directed to a sensor which generates a signalproportional to the light associated with that pixel element, the resultbeing a series of individual pixel output signals, together representingan input image frame. The entirety of that patent is incorporated byreference herein.

Such systems do not, however, provide image input and image outputcapability in a same shared resource system.

SUMMARY

One aspect of the invention relates to methods and apparatus forproviding imaging input and imaging output capability in a same sharedresource system.

In described embodiments, a spatial radiation modulator (SRM) unitcomprising an array of individually addressable and positionablereflecting surfaces receives input data in the form of electrical inputsignals for the purpose of determining how radiation incident on thesurfaces should be reflected to form respective pixel elements of anoutput image for display onto a display screen, printer, or otherimaging surface or target. The spatial modulator unit may, in someembodiments, take the form of a deformable minor device having an arrayof individually addressable and positionable specular surfaces thatrespectively reflect light incident from one or more output lightsources via an output imaging path, such as an optical lens path, toform a composite output image on an imaging plane. At least some of thesame specular surfaces are individually repositionable to respectivelyreflect radiation incident from an input image field of view of the sameSRM unit onto a camera or other radiation sensor along an input imagingpath to provide electrical output signals representative of a viewedinput image. In some embodiments, at least some portions of the inputimaging path and the output imaging path may utilize common resources.The radiation incident from the input image field of view may take theform of light or, alternatively, may take the form of other radiation,such as radiation at a low-terahertz frequency.

In a non-limiting example embodiment, described in greater detail below,a system for providing imaging input/output using a shared spatialmodulator uses a deformable mirror device (DMD) unit having an array ofspecular surface elements which are individually addressable andswitchable from positions reflecting light incident from a light sourceonto respective pixel locations of an imaging plane of an outputdisplay, to positions reflecting radiation from an input image field ofview to a photosensitive element of a camera or other photosensor, forgenerating electrical output signals proportional to intensity or othercharacteristic parameter of the received radiation. In one form, theinput image radiation is radiation received from an imaged subjectlocated between the DMD unit and the output image imaging plane. Inanother form, the input imaging radiation is radiation received from animaged subject located from the DMD unit beyond the output image imagingplane.

The shared spatial modulator input/output imaging system may be used invarious applications, such as disclosed, for example, in the variousProvisional Applications from which this application claims priority andthe entireties of which have been incorporated herein.

For so-called backlighting arrangements, in which an output image isdisplayed on one side of an imaging display surface for viewing from anopposite side of such surface, a switchable diffusion screen may beemployed for coordinated switching of the diffusion characteristic inblanket or pixel-selectable way with respect to the input imaging pathfor the capture of the input image.

The SRM unit may comprise lenses and other optics elements in the imageprojection and image reception paths, at least some of which may becommon to both the output and input imaging paths.

The SRM will typically be comprised of an array of reflective pixelelements, which are individually addressable and switchable, such that aseries of pixel-reflected light or other radiation beams are reflectedfrom the SM as different pixel elements are switched. A defocusing lensfocusing said pixel-reflected light onto a single-cell photosensor. TheSRM is in communication with timing means for controlling the switchingof its pixel elements, such that individual pixel elements representingimage pixels are addressed in synchronization according to the desiredcharacteristics of the image projection or capturing modes.

The unit may be used in digital or analog systems, for printing ordisplay applications with equipment that is remote or in-line with theradiation focusing and control elements. For digital applications, theoutput signals of the input imaging modes may be digitized and may besubjected to digital signaling processing or stored.

In both the output imaging and input imaging processes, the SRM enablesdifferentiation among individual pixel elements. In certain advantageousembodiments, information received from the input image process may beused to control one or more parameters of the output image process.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an arrangement of an embodiment illustrating principles ofthe invention.

FIG. 2 shows a modified form of the arrangement of FIG. 1.

FIG. 3 shows another modified form of the arrangement of FIG. 1.

FIGS. 4 and 5 show components of a spatial radiation modulator usable inthe arrangements of FIGS. 1-3.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

Described example embodiments are presented by way of non-limitingillustration of some of the many ways the claimed invention may beimplemented. For purposes of simplicity only, although other types ofspatial modulators may also be utilized, the example embodiments areshown in the context of systems utilizing spatial modulators of thedeformable mirror device type, such as the DLP™ spatial light modulatorscommercially available from Texas Instruments Incorporated, Dallas, Tex.

In an image projection (image output) application mode, a spatialradiation modulator (SRM), such as a DLP™ spatial light modulator 130shown in FIGS. 1-3, receives light from an illumination source 110incident on an array of individually addressable and positionablespecular reflecting elements. A controller 112 is responsive toelectrical input signals to set the positions of the individualreflecting elements to reflecting (“on”) or non-reflecting (“off”)positions corresponding to whether respective corresponding pixellocations in an output image should be illuminated or not illuminated.The SRM 130 then directs the modulated light, with the assistance ofvarious lenses and other optical devices 140 in an output imagingoptical path, onto an imaging plane of a display 150 (or other imagingsurface or target) for formation of an output image. The display 150 maybe an opaque surface such as a screen in a conference room, or may be atransparent or translucent surface such as a display screen of acomputer monitor.

In an image capture (image input) application mode, the SRM 130 receiveslight from an image capture subject 190 incident on individual ones ofthe reflecting elements of the same array for corresponding individualelement reflections to a camera or other sensor element 180 via an inputimaging radiation path that may include various lenses or other elements140 common with the output imaging path and/or may include other lensesor elements 120. An imaging unit 114 cooperates with the sensor element180 to receive and buffer electrical output signals corresponding to theimaging information received by the sensor 180, which are thencommunicated for further signal processing to a processing unit 118,which may include one or more digital signal processors, microprocessorsand/or other signal processing components.

In an advantageous form of the illustrated implementation, theprocessing unit 118 and controller 112 may be communicated so thatinformation obtained from a captured input image may be used for controlof one or more parameters of the output image process. For example, theprocessing unit 118 may include instructions for analyzing image datareceived from the imaging unit 114 to identify gestures of a humansubject interacting with an output image displayed on the display 150.The identified gestures may then be used to cause a response, such as bycausing the controller 112 to shift a position, enlarge a portion, openan image window, etc. in the displayer output image.

An input unit (or input/output unit) 122 is connected for human ormachine interface purposes to the input/output imaging system. Forexample, the input unit 122 may take the form of a keyboard and/or mousefor interacting with the controller 112 and/or processing unit 118. Theinput unit 122 may also provide porting for input/output electricalsignal communication from/to external processing or other electroniccircuitry.

FIG. 1 illustrates the case in which an output image is projected viaindividually addressable and positionable specular reflecting elementsin a DMD minor array onto a screen or other display surface 150 using afront projection mode. Such arrangement is typical of conference roomoverhead image projection systems, wherein an output image is projectedonto the front of an opaque screen. In such situations, it is notuncommon for a human subject (viz., a presenter) 190 to stand in frontof the screen and gesture to portions of the projected image (viz., inconnection with points made in a

In accordance with features of the described embodiment, an image of thehuman subject may be captured at imaging unit 114 using the describedimage input process, and the image subject may be analyzed in signalprocessing at processing unit 118 (or externally via communication frominput/output unit 122). Features or actions of the subject 190 may thenbe identified and used for providing input to controller 112, formodification of the image output process or taking other actionresponsive thereto. For example, an identification may be made of agesture of the subject 190 and the output imaging process can becontrolled to respond to the identified gesture to modify the displayedimage or modify one or more parameters of the displayed image (viewingarea, size, highlighting, animation, etc.). There is no requirement thatthe subject be in contact with the display screen 150.

The arrangement of FIG. 1 may also be used for teleconferencing inaddition to/or as an alternative to gesture recognition. In sucharrangement, the image of the human subject 190 can be captured andtransmitted to a remote location.

Advantageously, the illustrated image input radiation path may includethe same elements 140 from the subject 190 to the SRM 130 as the imageoptical path, thereby providing an image capture field of view for theinput image sensor 180 that may partly or completely overlap or coincidewith the projection field of view for the projected output image. Suchfield of view overlap or coincidence has advantages over, for example,video conferencing systems which employ cameras and projectors havingdifferent vantage points.

FIG. 2 illustrates the case in which an output image is projected viaindividually addressable and positionable specular reflecting elementsin a DMD mirror array onto a display surface 150 using a rear projectionmode. Such arrangement is typical of rear projection screens such asused as peripheral display devices for personal computers and the like.The same arrangement is used for rear projection advertising displays,large format televisions, etc. Here the output image is projected ontothe back of a, typically, partially transparent or translucent screen150. In such situations, a human subject (viz., a PC user, presenter orviewer) 190 is often positioned in front of the screen (that is, on theside opposite to the side onto which the image is projected).

The elements of the input/output imaging system of FIG. 2 may be largelysimilar to those for the arrangement shown in FIG. 1. One difference maybe the inclusion in a rear projection system of a diffusion screen 220,to provide a diffusing surface at the imaging plane. The presence of thediffusing surface may be an obstacle to the imaging input of a subject190 located on an opposite side of the screen 150 from the side on whichthe output image is formed. This can be overcome by providing aswitchable diffuser 220 whose switching is cooperated with the operationof the image input and output modulation switching, such as byconnection to the controller 112. The diffuser 220 may be madeswitchable in its entirety, or on a separately addressable (viz., rowand column) individual pixel or segment location basis. In this way, thetiming of image capture can be synchronized with switching of thediffuser so that the diffuser is made clear (or effectively renderedclear) at least at locations through which input imaging of the subject190 is to be made. Again, there is no requirement that the subject be incontact with the display screen 150; where, however, the input imagingfunction will used only for imaging a subject or areas of a subject incontact with the display screen (such as, for example, in touch screendetection), considerations of correcting for diffusion may not apply.

One implementation of a locally switchable diffuser utilizes a localelectric field applied at a pixel location to deform a flexible membranesubstrate to bring it locally at selectively addressable pixel locationsinto contact with another substrate. The local deformation can bebrought about using a mechanism such as used in the Time MultiplexedOptical Shutter (TMOS) display methodology disclosed in the issuedpatents of Unipixel™ Displays, Inc., The Woodlands, Tex. (e.g., U.S.Pat. Nos. 5,319,491; 7,042,618; 7,092,142; 7,256,927; 7,486,854;7,449,759; 7,515,326; 7,522,354; 7,535,611), the entireties of all ofwhich are incorporated herein by reference. In one embodiment usableherein, the membranes could be arranged to provide a normally diffusingpixel location onto which a corresponding pixel of the output image canbe displayed and a non-diffused pixel location providing a clear opticalpath upon selective application (or non-application) of a local electricfield. The selectivity of the clear pixel locations can be synchronizedwith the positioning of the movable surfaces of the corresponding pixelsof the SRM 130, so that diffusion is presented for the output imagingprocess and transparency is presented for the input imaging process.Other arrangements are possible.

It will be appreciated that imaging through a diffusion screen 220 maycreate artifacts or effects that can also be compensated for by applyingimage correction algorithms in software, as for example by executing aset of non-transient program instructions in the signal processingexternally or at processing unit 118, or by applying image correctionalgorithms in hardware, or partially in software and partially inhardware, used programmable logic gates or other logic arrangements.

FIG. 3 shows a rear projection arrangement similar to that of FIG. 2,but which avoids issues associated with diffusion by utilizing radiationfor which the diffuser 220 is effectively transparent. Such radiationmay be found in parts of the electromagnetic spectrum, for example,separated from the parts commonly considered to be in the visible lightrange. An advantageous candidate for this purpose is electromagneticenergy in the terahertz frequency range. Such radiation has thecharacteristic that it can be directed by the same reflecting surfacesof, e.g., a DMD mirror array, yet will pass unobstructed through acommon diffuser 220.

As shown in FIG. 3, a source of terahertz frequency electromagneticradiation 182 can be positioned to be incident on a partially reflectingmirror 184 or the like, to irradiate the subject 190 along the sameradiation path as the optical path used for the output image. Theterahertz frequency characteristic enables the radiation to be directedby the individually switchable elements of the array 130, then passthrough the effectively transparent diffuser 220. The radiation can thenreflect off the subject 190, back through the diffuser 220 and becaptured by a sensor 180 which is sensitive to radiation in theterahertz frequency range. The captured radiation image can then beanalyzed to identify gestures, etc.

This of the projection having common elements with the image, andbecause the image input system uses shared elements with the imageoutput Because the subject is imaged through the same ed region,location size, consider, e.g., the location in an of a pointing fingercould be identified input process as part of surface located in front ofsurface onto which the output image is directed with the screen orenlacause a response in the is responsive to electrical input signals toset the positions of the individual specular reflecting elements toreflecting (“on”) or non-reflecting (“off”) positions corresponding tothe illumination intensities of respective corresponding pixel locationsin an output image. The SRM 130 then directs the modulated light, withthe assistance of various lenses and other optical devices in an outputimaging optical path, for formation of an output image onto an imagingplane (or other imaging surface or target) of a display 150, which maybe an opaque surface, such as a wall screen in a conference room, or atransparent or translucent surface, such as a display screen of acomputer monitor. information received from the input image process maybe used to control one or more parameters of the output image process.

The embodiment is described in terms of implementing SRM 130 with adeformable mirror device (DMD). However, the invention is not limited tothe use of DMDs for SRM 130 and may be implemented with other types ofSRMs. A common characteristic of a suitable SRM device is the ability toreflect light from pixel elements that are individually addressable.

DMD 130 is a deformable mirror device (DMD), which is used to reflectpoints of light incident from illumination source 110 or incident fromsubject 190 as pixels, on a selectable pixel by pixel basis. DMDs are atype of spatial light modulator having an array of reflective pixelelements. Each reflective element represents a pixel element, and eachis electronically addressable and capable of separate mechanicalmovement in response to an electrical input. For display, each pixelelement is switched so that it is tilted to one of a number ofpositions.

An example DMD device is the DLP™ device manufactured by TexasInstruments Incorporated, Dallas, Tex., in which each pixel element isassociated with a memory cell and may be individually addressed. OtherDMD devices may be used, and are characterized by various types ofarchitectures. The mirror elements may be moveable by means oftorsion-beam or cantilever supports, or may be elastomer or membranedesigns. Addressing may be achieved by an e-beam input, optically, or byintegrated circuits.

FIGS. 4 and 5 illustrate a movable pixel element 41, representative ofthe pixel elements that comprise the pixel array of the DMD mirror array130. When pixel element 41 is placed in one position, it is directed(oriented) to reflect light incident from illumination source 110through the projection lens 140 and onto the imaging surface 150. Whenpixel element 41 is placed in another position, it is directed toreflect light (or other radiation) incident from the subject 190 throughthe lens 140 and onto the sensor 180.

The pixel element 41 of FIGS. 4 and 5 is of a torsion-beam design, wherea thick reflective beam, i.e., minor 42, is suspended over an air gapand connected between two rigid supports by two thin torsion hinges 43 aand 43 b that are under tension. When an address electrode 44 a or 44 b,underlying one-half of minor 42, is energized, the torsion hinges 43 aand 43 b are twisted and mirror 42 rotates about the axis of the twohinges 43 a and 43 b.

The movement of minor 42 is shown in FIG. 5. Minor 42 moves about anaxis from a position shown by the dotted line 45 a (viz., position forreflecting light from illumination source 110 to display screen 150) toa position shown by the dotted line 45 b (viz., position for reflectinglight from subject 190 to sensor 180). In one position, the edge ofminor 42 touches a landing electrode 46 b; and in another position, theedge of mirror 42 touches a landing electrode 46 a. Minor 42 is rotatedbetween positions by applying appropriate opposite voltages to addresselectrodes 44 a, 44 b. A differential bias is applied to mirror 42through electrode 47.

The torsion-beam pixel of FIGS. 4 and 5 is only one type of pixelarchitecture, and many other architectures are possible. These aredistinguished by characteristics such as their deformation mode, pixelshape, and the hinge support architecture. However, for purposes ofapplication herein, any sort of architecture is satisfactory thatenables independent movement of the pixel elements.

DMD mirror array 130 is addressed pixel by pixel, moving individualminors 42 between their individually selected positions as appropriateto provide the desired image output and image input functions. Thecontrol for addressing and switching of the pixel elements of the array130 in the illustrated arrangements is provided by controller 112. Suchcontrol may, however, be provided by a separate array control element.Controller 112 also includes means for generating timing signals for thecoordination of the output and input imaging functions. U.S. Pat. No.5,214,420, “Spatial Light Modulator Projection System with RandomPolarity Light,” the entirety of which is incorporated herein byreference, describes a method of addressing the individual pixels. Usingsuch method, controller 112 includes a decoder that receives a rowaddress and decodes the address to select the desired row of pixels.Similar means may be used to select the column of a pixel to beaddressed. If pixels are to be addressed on a column-by-column,row-by-row basis, simple position-shifting techniques may be used toenhance speed of operation.

Other Embodiments

Those skilled in the art to which the invention relates will appreciatethat the described example embodiment and its various arrangements isillustrative of just some of the many possible embodiments andmodifications of embodiments that can be implemented within the scope ofthe claimed invention.

1. (canceled)
 2. The method of claim 3, wherein the spatial radiationmodulator is a deformable mirror device.
 3. A method of displaying afirst image at a display location and capturing a second image of asubject located proximate the display location utilizing radiation pathshaving at least partially common paths including at least one spatialradiation modulator; wherein the radiation includes electromagneticradiation having a terahertz frequency.
 4. The method of claim 3,wherein the method further comprises controlling subject matter,characteristics or display parameters of the first image in response toinformation determined by capture of the second image.
 5. A system fordisplaying and capturing images, comprising: a radiation source; aspatial radiation modulator positioned to receive radiation from theradiation source and direct the radiation for display of at leastportions of an image to a display location, responsive to first positionsettings of individually addressable radiation reflecting members; and aradiation sensor positioned to capture at least portions of an image ofa subject positioned adjacent the display location, responsive to secondposition settings of the individually addressable radiation reflectingmembers; wherein the image is displayed on one side of an imagingdisplay surface for viewing from an opposite side of the surface; andthe system further comprises a switchable diffusion screen forcoordinated switching of a diffusion characteristic for capture of animage of a subject on the opposite side of the surface. 6-7. (canceled)8. A system for displaying and capturing images, comprising: a radiationsource; a spatial radiation modulator positioned to receive radiationfrom the radiation source and direct the radiation for display of atleast portions of an image to a display location, responsive to firstposition settings of individually addressable radiation reflectingmembers; and a radiation sensor positioned to capture at least portionsof an image of a subject positioned adjacent the display location,responsive to second position settings of the individually addressableradiation reflecting members; wherein the system comprises a book orsimilar medium with a capability for displaying moving pictures; theradiation source includes a light illumination source; the spatialradiation modulator comprises a multiplicity of individually addressablemirror members for directing the light onto or through an area of amedium having multiple pages responsive to first position settings ofthe minor members; and the radiation sensor comprises a camera having afield of view for capturing an image of a viewer of the projected imageresponsive to second position settings of the minor members; and furthercomprising circuitry for shifting the mirror members between the firstand second position settings responsive to action of the viewer detectedin response to information obtained from the captured image of thesubject.
 9. The system of claim 8, wherein the circuitry furthercomprises circuitry for selecting a displayed image responsive to theaction of the viewer.
 10. The system of claim 9, wherein the circuitryfurther comprises circuitry for adding at least a portion of an image ofthe viewer to the displayed image
 11. A system for displaying andcapturing images, comprising: a radiation source; a spatial radiationmodulator positioned to receive radiation from the radiation source anddirect the radiation for display of at least portions of an image to adisplay location, responsive to first position settings of individuallyaddressable radiation reflecting members; and a radiation sensorpositioned to capture at least portions of an image of a subjectpositioned adjacent the display location, responsive to second positionsettings of the individually addressable radiation reflecting members;wherein the system comprises a display device with an integral securitymonitor; the radiation source includes a light illumination source; thespatial radiation modulator comprises a multiplicity of individuallyaddressable minor members for directing the light onto a display surfaceresponsive to first position settings of the minor members; and theradiation sensor comprises a camera having a field of view for capturingan image of a subject of a viewer beyond the display surface responsiveto second position settings of the mirror members; and furthercomprising a data source for generating imaging pixel data relating toflight information; and control circuitry for shifting the mirrormembers between the first and second position settings insynchronization for display of the flight information and for capture ofinformation relating to the viewer of the flight information on thedisplay.
 12. The system of claim 11, wherein the control circuitryfurther comprises data signal processor adapted to identify a threatresponsive to information obtained from the captured image of thesubject.